There has been known an imaging system configured to generate a stereoscopic image or free-viewpoint image based on multiple images obtained by imaging the same subject from multiple viewpoints using a camera. With this imaging system, there is performed processing to calculate distance information (depth information) of a subject from disparity that is relative shift amount between images using the principle of triangulation and to generate an image viewed from an optional virtual viewpoint based on distance information thereof.
In the event of calculating disparity between two images, generally, detection of corresponding points between two images is performed. As a method for detecting corresponding points, there have been known SAD method employing Sum Of Absolute Difference (SAD), or SSD method employing Sum of Squared Difference (SSD). With these methods, first, of two images, one of the images is taken as a base image, and the other is taken as a reference image. Also, a region including a point of interest within the base image is set as to this point of interest, and multiple regions having the same size as the region thereof are also set in the reference image. Next, a degree of correlation between the region within the base image and each region within the reference image is obtained by block matching, and a center point of a region within the reference image having the highest degree of correlation is detected as a corresponding point corresponding this point of interest. Detection of a corresponding point is performed by searching above an epipolar line. The epipolar line is a straight line where an epipolar plane, which is a plane including two viewpoints and a target point in space or above an object, intersects each of the images.
FIG. 23 is a schematic diagram illustrating an epipolar plane and epipolar lines.
Description will be made regarding a case where a target point 3 in space is shot by a first camera 1 which is a left camera and a second camera 2 which is a right camera, as an example. In this case, epipolar lines 7 and 8 are intersections between a triangular plane (epipolar plane) 4 made of three points of the target point 3 in space and two viewpoints (the centers of lenses of the first camera 1 and second camera 2), and three line segments connecting these, and an image plane 5 shot by the first camera 1 and between the triangular plane 4 and an image plane 6 shot by the second camera 2. A corresponding point within the other image corresponding to a point of interest within one of the images exists above an epipolar line, and accordingly, the corresponding point can be detected by searching above the epipolar line.
FIG. 24 are schematic diagrams illustrating a case where the first camera 1 and second camera 2 have been disposed so that the optical axes of the first camera 1 and second camera 2 become parallel. In this case, the epipolar lines 7 and 8 becomes horizontal straight lines having the same height in both image planes, and accordingly, a corresponding point can be searched above the same line in both images.
However, image shift may occur between the two images due to attachment precision of the two cameras, or distortion of lenses, or the like, and accordingly, the epipolar line in each camera does not necessarily become the same line within the two images. In the event that shift is not small, a corresponding point cannot be found, and regions where accurate disparity cannot be calculated increase. In order to accurately calculate disparity, image shift needs to be corrected, but a simple method wherein the entire screen is moved for adjustment, for example, is low in correction precision, and it is difficult to sufficiently correct image shift caused due to attachment precision of a camera, or distortion of a lens, or the like.
On the other hand, in PTL 1 (Japanese Unexamined Patent Application Publication No. 2001-82955), a device has been disclosed wherein rotational shift or translational shift of images is corrected in real-time using affine transformation which is geometric coordinate transformation. Also, in PTL 2 (Japanese Unexamined Patent Application Publication No. 2003-4442), a device has been disclosed wherein a second pattern having the highest correlation with a first pattern within one of images is detected out of multiple horizontal and vertical lines within the other image, and disparity is calculated from first and second position information of the first and second patterns. According to this device, search of a corresponding point is performed above horizontal lines and vertical lines within the image, and accordingly, incorrect ranging due to image shift can be prevented even without correcting the entire image.